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INITIAL  EVALUATION  OF  PRINCIPLES  FOR 
GRAPHICAL  DISPLAYS  IN  MAINTENANCE  PROBLEM  SOLVING 


INTRODUCTION 

The  purpose  of  this  work  is  twofold.  The  first  goal  is  to  develop  principles  to  guide  the 
transformation  of  hardcopy  formats  of  technical  information  to  electronically  displayed  formats. 
The  end-users  of  the  displays  will  be  maintainers  of  complex  systems.  The  primary  motivation 
in  this  transfer  is  the  need  to  go  from  large  blueprint  sized  hardcopy,  often  called  C  size,  to  small 
CRT  display  sized  images.  These  external  representations  of  maintenance  information  should 
support  the  internal  representation  and  processing  of  the  maintainer.  There  are  three  aspects  of 
the  use  of  this  information  that  must  be  understood  before  such  principles  can  be  developed. 

One  is  the  nature  of  the  device  being  maintained.  This  component  drives  the  other  two  by 
providing  components,  actions,  and  relationships  that  must  represented  or  carried  out.  The 
second  aspect  is  the  nature  of  the  information  available  and  tasks  carried  out  in  the  maintenance 
environment.  Three  is  the  nature  of  the  knowledge  and  cognitive  processes  of  the  maintainer. 
The  relationships  among  these  are  depicted  in  Figure  1. 

Figure  1.  Maintenance  Domain  Figure  goes  about  here 


The  second  goal  of  this  work  is  to  evaluate  these  display  principles  in  a  realistic  context  with 
maintainers  of  an  operational  system.  The  principles  were  used  to  generate  displays  in  an  actual 
maintenance  domain.  Then  the  displays  were  given  to  maintainers  in  paper  form  to  use  in  real 
versions  of  their  maintenance  tasks.  Since  this  analysis  and  evaluation  resides  in  the  larger 
context  of  attempting  to  understand  and  aid  maintainers  of  complex  systems,  and  since  the  work 
is  being  done  with  real  maintainers,  the  results  will  be  generalizable  to  a  variety  of  maintenance 
domains. 

BACKGROUND 

Current  work  aimed  at  understanding  the  use  of  external  representations  in  problem  solving 
of  any  type  might  be  summarized  best  by  turning  to  Figure  2  which  is  a  much  simplified 
illustration  of  the  general  processes  involved  in  cognition.  This  figure  depicts  a  visual 
perception  or  understanding  process  (B)  by  which  information  is  taken  from  an  external 
representation  (A)  and  produced  in  the  form  of  an  internal  representation  (C)  that  reflects  the 
external  world  (veridical  representation).  It  next  depicts  a  concept  formation  process  (D)  by 
which  the  initial  representation  (C)  is  either  translated,  transformed,  or  re -constructed  as  a 


relevant  internal  representation  (E).  Finally,  it  depicts  a  problem  solving  process  (F)  by  which 
the  relevant  internal  representation  (E)  is  matched  to  some  rule,  model,  or  other  form  to  generate 
new  internal  representations  for  continued  processing  or  to  generate  an  external,  or  motor, 
action. 

Figure  2.  General  Cognitive  Process  goes  about  here 


The  work  on  use  of  external  representations  for  perception  and  performance  has  produced 
many  models  of  particular  cognitive  processes.  One  expects  these  models  could  predict  how 
information  taken  in  the  form  of  an  external  representation  would  be  used  as  an  internal 
representation  for  problem  solving.  However,  there  is  a  crucial  gap  in  the  work  on  these  models. 
Researchers  and  theorists  at  the  perceptual  input  level  (e.g.,  Feldman,  1984;  Kosslyn,  Flynn,  & 
Amsterdam,  1989;  Marr,  1982;  McClelland  &  Rumelhart,  1981)  tend  to  model  representation 
and  processing  from  external  representation  to  an  internal  representation  (i.e.,  from  A  to  C). 

They  assume  that  the  internal  representation  veridically  reproduces  the  external  world. 
Researchers  and  theorists  at  the  next  level  (e.g.,  Kieras,  1988;  Larkin,  1989;  Larkin  &  Simon, 
1987)  assume  an  appropriate,  relevant  representation  at  the  problem  solving  process  (i.e.,  from  E 
to  F).  They  do  not,  however,  account  for  how  the  representation  arrived  in  suitable  form. 

In  terms  of  the  general  cognitive  process  described  in  Figure  2,  little  if  any  work  explicitly 
covers  the  process  of  getting  a  representation  from  the  veridical  internal  representation 
(reproduction  of  the  external  world)  to  the  internal  representation  that  is  relevant  to  the 
maintainer's  problem  (i.e.,  from  C  to  E).  Yet,  this  is  precisely  where  the  problem  lies  for  making 
external  representations  useful  for  problem  solvers.  The  closest  work  falls  under  the  rubrics  of 
concept  formation,  concept  coherence,  classification,  or  categorization  (e.g.,  Barsalou,  1984; 
Osherson  &  Smith,  1982).  Several  researchers  and  theorists  in  these  areas  are  working  on 
computational  models  of  concepts  (e.g.,  Lebowitz,  1986;  Michalski  &  Stepp,  1983;  Smith, 
Osherson,  Rips,  &  Keane,  1988);  but,  no  one  in  those  areas  seems  to  be  developing,  applying,  or 
testing  the  models  in  goal-directed  activities  such  as  problem  solving  in  an  attempt  to  bridge  this 
gap  in  the  general  view  of  cognitive  processing.1 

For  example,  Larkin  and  Simon  (1987)  apply  a  recognition  mechanism  which  assumes  the 
user  has  a  more  or  less  exact  representation  already  stored  of  what  they  are  seeing.  Kieras 

1  There  has  been  some  work  by  theorists  such  as  Rumelhart  and  McClelland  (1986)  on  concept  acquisition,  and  by  Kotovsky  and 
Fallside  (1989)  on  transfer.  We  suspect  Kotovsky  and  Fallside  would  disagree  with  our  view  since  they  conclude  that  “it  is  the 
internal  representation  of  a  problem  that  determines  transfer,  and  this  representation  can  operate  independently  of  the  stimulus 
features.  (p,105)“.  Our  view  is  that  the  formation  of  an  internal  representation  or  concept  always  depends,  at  some  level,  on  the 
stimulus  features  of  the  problem.  That  is,  the  mainlaincr  creates  or  adapts  the  internal  map  to  address  the  problem  at  hand. 


(1988)  improves  on  this  slightly  by  invoking  retrieval  mechanisms  thereby  implying  some 
construction  or  transformation  process  may  occur.  Recognition  or  retrieval  is  certainly  one  way 
our  processing  system  works  and  is  very  efficient  computationally.  However,  recognition  and 
retrieval  work  alone  only  in  a  limited  number  of  cases  and  conditions.  Complete  bridging  will 
require  additional  mechanisms  for  transformation  or  construction  of  representations  during  the 
process  of  forming  the  relevant  internal  representation  for  the  problem  solving  process. 

CONCEPT  FORMATION  PERSPECTIVE 

We  need  to  determine  how  to  transform  very  large  schematic  diagrams  into  useful  computer 
screen  size  (5-9  inch,  12.70  -  22.86  cm  diagonal)  displays  for  use  by  maintained  of  complex 
systems.  Shrinking  the  diagrams  to  screen  size  is  an  inadequate  solution  since  the  complexity  of 
the  current  diagrams  does  not  translate  well  into  very  small  screens.  Instead  we  must 
reconstruct  the  diagrams  so  they  are  aligned  with  the  information  requirements  of  the  maintainer. 
These  information  requirements  are  likely  to  be  dynamic,  based  on  the  wide  variance  in  user 
experience  and  nature  of  the  maintenance  task. 

During  problem  solving  (in  this  case  maintenance  problem  solving),  the  problem  solver 
seeks  information  from  various  sources.  The  context  of  the  problem  guides  the  search  for  that 
information.  The  problem  solver  must  "transform"  externally  provided  information  into 
internally  maintained  conditions  for  problem  solving  (whether  it  is  rule-firing  or  network- 
activation;  whether  it  is  procedure-based  or  model-based).  We  hypothesize  that  there  is  an 
active  concept  formation  process  which  produces  the  transformation  from  external  representation 
to  relevant  internal  representation. 

One  plausible  mechanism  in  this  concept  formation  process  is  structure-mapping  (Gentner, 
1983,  1989).  Gentner  has  proposed  structure-mapping  as  a  computational  mechanism  by  which 
analogous  concepts  are  recognized  or  formed.  In  the  case  of  maintenance  problem  solving, 
structure-mapping  as  a  concept  formation  process  would  take  externally  supplied  information 
and  try  to  map  it  to  the  set  of  possibilities  that  are  the  internally  maintained  conditions  for 
problem  solving  in  the  current  context.  When  there  is  not  a  direct  mapping  then  a  next  best  or 
analogous  mapping  could  be  made  so  the  problem  solving  process  could  continue. 

Structure-mapping  can  be  operationally  defined  as  judging  the  similarity  of  a  target  stimulus 
to  a  base  stimulus  through  the  dimensions  of  a  similarity  space.  Gentner  defines  a  similarity 
space  composed  of  two  dimensions.  These  dimensions  are:  1)  shared  features  and,  2)  shared 
structure  (relations  among  features).  This  view  has  a  good  foundation  starting  with  the  work  of 


Tversky  (1977).  However,  there  is  a  fundamental  weakness  in  its  development.  The  definition 
of  structure  mapping  provides  no  means  for  defining  what  are  features  (and  how  to  count  them) 
and  what  are  relations  (and  how  to  count  them).  We  propose  that  the  definitions  of  features  and 
relations  are  dependent  on  the  purpose  of  defining  them.  If  our  purpose  is  to  use  features  and 
relations  as  a  way  of  defining  the  similarity  space  between  base  and  target  stimuli  in  a 
maintenance  domain  then  there  must  be  an  objective  means  for  defining  these  entities.  One 
means  is  to  use  the  informational  and  behavioral  characteristics  of  operational  and  maintenance 
environments  as  defined  by  Rasmussen  (1983,1986). 

Rasmussen  suggests  that  operational  and  maintenance  problem  solving  is  characterized  by 
three  general  classes  of  cognitive  activities  which  operate  on  representations  of  information.  The 
classes  of  cognitive  activities  are  characterized  by  skill-based,  rule-based,  and  knowledge-based 
problem  solving.  Representations  are  characterized  by  their  location  in  a  space  defined  by  two 
dimensions:  aggregation  and  abstraction.  According  to  this  view,  different  representations  are 
used  differently  for  different  classes  of  activities  depending  on  the  information  available  and  the 
sequence  of  processes  leading  to  the  current  state.  Furthermore,  those  representations  may  be 
operationally  defined  by  their  position  in  the  aggregation  by  abstraction  space.  Operationally, 
this  means  that  structure-mapping  as  a  cognitive  process  should  trigger  different  overt  behaviors 
depending  on  the  match  between  external  information  and  internal  representations.  Both  sets  of 
these  representations  can  be  defined  according  to  their  position  in  the  aggregation  by  abstraction 
space.  This  allows  us  to  manipulate  external  representations  according  to  this  space. 
Consequently  we  should  be  able  to  determine  the  effect  of  different  representations  on  the 
maintenance  problem-solver's  cognitive  process  and  subsequent  performance. 

We  propose  the  different  uses  of  external  representations  are  due  to  the  differing  information 
required,  concepts  formed,  and  cognitive  problem  solving  processes  engaged.  In  particular,  we 
propose  that  the  active  concept  formation  mechanism  of  structure  mapping  is  the  primary 
determiner  of  the  utility  of  external  representations.  One  means  for  supporting  structure¬ 
mapping  is  to  define  external  representations  that  take  advantage  of  the  capabilities  of  the 
mechanism.  Defining  external  representations  according  to  levels  of  abstraction  and  aggregation 
at  which  the  representations  within  the  current  problem  solving  process  exist  should  facilitate  the 
structure-mapping  from  external  representation  to  relevant  internal  representation.  If  we  develop 
a  range  of  displays  which  covers  the  complete  abstraction  by  aggregation  space  then  we  should 
support  the  problem  solver  no  matter  the  level  at  which  he  or  she  is  working. 

In  the  following  studies,  we  explore  the  abstraction  by  aggregation  information  space  and  the 
cognitive  processes  relevant  to  maintenance  problem  solving  in  a  real-world  domain.  In  the  first 
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study  we  developed  a  cognitive  framework  for  maintenance  problem  solving  and  concept 
formation  based  on  observation  of  a  subject  matter  expert  solving  maintenance  problems.  Using 
the  results  of  the  first  study  as  a  guide,  we  developed  initial  principles  for  displays  based  on 
information  needed,  concepts  formed,  and  cognitive  processes  employed  during  maintenance 
problem  solving.  Then,  for  the  second  study  we  used  these  principles  to  further  refine  the 
framework,  explore  the  relationship  between  concept  formation  and  maintenance  problem 
solving,  and  develop  and  test  displays  to  be  used  during  maintenance  problem  solving. 

STUDY  ONE 


OBJECTIVES 

The  objective  of  this  study  was  to  develop  a  general  domain  model  for  maintenance  problem 
solving  for  the  SH-3  helicopter  bladefold  system.  Development  of  an  initial  framework  for 
cognition  in  maintenance  problem  solving,  and  an  initial  aggregation  by  abstraction  framework 
for  information  needed  and  concepts  constructed  during  maintenance  problem  solving  were  also 
objectives. 

The  general  definitions  evolve  from  Rasmussen  (1983, 1986).  Constrasting  Rasmussen's 
rule-based  and  knowledge-based  behavior  with  current  cognitive  views  of  problem  solving  (e.g., 
Greeno  &  Simon,  1988;  Jones  &  Langley,  1988),  we  are  led  to  reconsider  the  definitions  of  these 
classes  of  problem  solving.  According  to  traditional  cognitive  views,  rule-based  problem 
solving  as  a  procedure  based  problem  solving  process  can  stand  as  defined  by  Rasmussen. 
Knowledge-based  problem  solving,  however,  might  be  defined  better  as  model-based  problem 
solving  which  corresponds  more  closely  to  views  of  problem  solving  as  a  form  of  model 
execution  to  generate  and  test  hypotheses  (White  &  Frederiksen,  1988,  1989). 

For  our  purposes,  an  important  contributor  to  the  variability  in  the  different  kinds  of  problem 
solving  is  the  match  of  the  information  provided  for  task  performance  to  the  information  needed 
for  the  cognitive  activities  required  for  performance.  Rasmussen  defines  the  relevant  dimensions 
of  this  information  as  level  of  aggregation  (kinds  of  features)  by  level  of  abstraction  (kinds  of 
relations).  The  match  is  made  through  a  structure-mapping  mechanism  according  to  a  similarity 
judgment. 

We  propose  that  Rasmussen's  definitions  for  aggregation  by  abstraction  are  appropriate 
operational  definitions  for  Gentner’s  similarity  space  definitions;  and,  that  behaviors  generated 
by  rule-based  and  model-based  problem  solving  are  differentially  affected  by  information 
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displays  represented  at  different  places  in  this  two-dimensional  space.  We  expected  to  see  this  in 
observations  from  this  study. 

METHOD 

The  general  method  was  a  think-aloud  protocol  study  in  which  a  subject  matter  expert  was 
asked  to  work  through  3  problems  in  each  of  two  ways  (rule-based  problem  solving  and  model- 
based  problem  solving).  The  protocols  were  analyzed  for  actions  taken,  information  used,  and 
the  relationships  between  actions  and  information. 

Participant 

The  participant  (M)  was  an  aviation  mechanic  with  over  15  years  of  experience  performing 
and  teaching  maintenance  troubleshooting  in  various  environments.  Although  M  never  worked 
in  the  SH-3  helicopter  environment,  he  had  attended  the  training  school  for  SH-3  bladefold.  He 
also  served  as  the  knowledge  engineer  for  a  research  project  to  develop  an  intelligent  tutoring 
system  authoring  environment  which  used  the  SH-3  as  its  domain  example. 

Materials 

Materials  were  three  SH-3  bladefold  maintenance  problems,  the  SH-3  maintenance  manuals 
relevant  to  bladefold  maintenance  problems,  and  protocol  transcription  sheets. 

Procedure 

The  experimenter  (E)  had  M  think  aloud  while  working  each  of  three  problems  from  the 
training  course  in  which  he  had  participated.  During  the  protocols,  E  wrote  down  the  verbal 
material  that  M  generated2.  During  this  process,  E  tracked  all  actions  M  performed,  all 
information  sought,  all  symptoms  dealt  with,  and  all  solutions  obtained.  E  and  M  held  post-hoc 
discussions  to  clarify  the  protocols.  During  that  process,  E  and  M  also  identified  all  relevant 
objects  by  talking  through  and  examining  all  diagrams;  and,  identified  all  actions/tasks  by 
talking  through  and  examining  all  procedures.  Finally,  E  and  M  identified  all  sources  of 
information  by  talking  through  and  examining  the  manuals  as  they  were  used  in  the  problem 
solving  process. 


2 


These  are  not  verbatim  transcripts  of  M's  think  aloud  process.  They  are  a  capture  of  some  verbatim  quotes,  some  phrases,  and  some 
summaries  of  M's  process. 


6 


M  worked  each  problem  twice.  First  he  worked  it  using  the  fault  tree  provided  in  the 
maintenance  manual.  This  provided  the  rule-based  version  of  the  problem  solving  process.  The 
second  time  he  worked  the  problem  using  the  models  he  had  learned  in  the  training  course.  This 
provided  the  model-based  version  of  the  problem  solving  process.  This  process  yielded  6 
protocols.  Of  course  there  are  obvious  limitations  to  this  method  but  it  served  well  as  a  tool  to 
generate  the  initial  framework  to  test  later  on  experienced  maintainers. 

RESULTS 

The  results  of  this  study  are  split  into  two  components.  One  set  of  results  addresses  the  goal 
of  developing  a  cognitive  framework  for  maintenance  problem  solving.  The  other  set  of  results 
addresses  the  goal  of  illuminating  the  information  needed  and  concepts  formed  during  the 
maintenance  problem  solving  process,  especially  in  the  context  of  the  abstraction  by  aggregation 
information  space. 

Cognitive  Framework 

Data.  The  data  used  to  develop  the  initial  framework  were  the  statements  M  made  and  the 
diagrams  M  examined  during  each  problem  solving  session.  These  were  carefully  tracked  and 
recorded  so  they  could  be  used  to  reconstruct  M's  problem  solving  process  later.  After 
reconstructing  the  problem  solving  process,  E  and  M  went  through  each  protocol  to  guarantee  its 
correctness.  The  protocol  statements  were  then  used  to  identify  cognitive  tasks,  information 
used,  and  the  relationships  among  them. 

Findings.  Our  initial  analyses  of  the  protocols  collected  from  M  show  eight  tasks  that 
maintainers  perform  in  the  pursuit  of  their  duties.  They  are  listed  here  and  are  discussed  further 
in  the  following  paragraphs: 

1.  look 

2.  identify 

3.  match 

4.  reason 

5.  remember 

6.  locate 

7.  repair 

8.  test 

The  distribution  of  the  tasks  among  the  two  approaches  to  solving  the  maintenance  problem  and 
the  three  problems  is  shown  in  Table  1.  The  numbers  are  interesting  but,  given  that  they  were 
derived  from  one  maintainer,  should  not  be  given  too  much  weight.  The  only  striking 
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observation  from  the  table  is  the  high  frequencies  of  tasks  locate,  look,  remember,  and  test 
compared  to  the  others. 


Table  1.  Task  Distribution  goes  about  here 


The  task  of  look  refers  to  the  act  of  visually  acquiring  information  about  some  state  or 
relation.  In  reference  to  our  general  processing  model  (Figure  2),  this  action  reflects  the  visual 
understanding  process  transforming  an  external  representation  into  a  direct  perception  or 
veridical  representation.  For  example,  M  would  "look  in  the  technical  manual  for  problem 
condition  in  decision  tree."  Or,  M  would  "look  at  the  Blade  Fold  System  Location  Diagram  to 
find  information  about  the  location  of  PI  11."  Or,  finally,  M  would  "look  at  [a  diagram]  to  find 
the  parts  relevant  to  a  part  of  interest."  In  each  of  these  cases  M  is  examining  an  external 
representation  of  some  aspect  of  the  current  problem. 

The  task  of  identify  refers  to  the  formation  of  an  internal  representation  to  reflect  some  state 
of  the  external  world  or  the  problem  solving  process.  In  reference  to  Figure  2,  this  action 
reflects  the  concept  formation  process  transforming  from  C  to  E.  For  example,  identify  often 
referred  to  confirmation  of  the  problem  condition  that  needed  to  be  addressed  by  the  maintained 
In  such  cases  it  was  usually  referred  to  by  M  expressing  a  need  to  see  the  indications  of  the 
problem  for  himself.  For  example,  he  would  "identify  the  problem  condition  by  confirming  the 
reported  problem  using  a  procedural  checklist".  It  could  also  refer  to  confirmation  of  some 
device  element  as  being  part  of  some  component.  For  example,  M  reported  "identifying  the  limit 
switch  as  part  of  the  control  lock  cylinder".  In  either  case,  identify  refers  to  representing  the 
condition(s)  of  the  current  problem. 

The  task  of  match  refers  to  finding  an  equivalence  between  a  set  of  problem  conditions  and 
some  aspect  of  either  a  rule  or  a  model.  In  reference  to  the  general  cognitive  model,  this  action 
reflects  activity  in  the  problem  solving  process.  In  the  case  of  rules,  there  were  an  explicit  set  of 
rules  provided  for  the  maintainer  to  use  in  attaining  solutions  to  problems.  So  the  match  was 
between  the  problem  conditions  and  the  antecedent  clause  in  a  rule  (or  the  first  in  a  chain  of 
rules).  For  example,  M  matched  the  initial  condition  of  blade-positions-but-will-not-fold  with 
TROUBLE  #17  in  the  fault  tree.  In  the  case  of  models,  M  was  usually  trying  to  generate  a 
match  between  a  state  produced  by  some  causal  relationship  and  the  problem  conditions 
provided.  For  example,  M  matched  the  model  generated  blades -folded-and- no-lights-means- 
bad-light-OR-means-bad-connection-between-slip-rings  to  the  initial  conditions  of  blades- 
folded-bladefolded-light-does-not-go-on.  In  each  of  these  examples,  M  is  comparing  one 
representation  of  the  system  to  another  representation  of  the  system  in  search  of  a  match.  At  the 
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same  time,  though,  there  also  appear  procedural  rules  in  the  middle  of  this  model-based 
processing.  For  example,  while  reasoning  about  the  relation  of  limit  switches  to  blade  fold,  M 
matched  and  executed  a  procedure  for  a  test  of  limited  relevance  to  the  problem. 

The  task  of  reason  refers  to  a  process  by  M  that  appears  very  much  like  deductive  logic. 

This  action  also  refers  to  activity  in  the  problem  solving  process.  For  example,  M  used  his 
model  to  reason  as  follows: 

"...between  slip  ring  6  and  slip  ring  2  there  is  a  switch  for  each  blade.  The  blade  rests  on 
the  switch  when  the  blade  is  folded.  So,  either  a  switch  is  broken  or  a  wire  is  broken 
between  slip  ring  6  and  slip  ring  2.  So,  test  each  switch  and  the  wires  between  each 
switch." 

In  this  example,  as  in  a  few  others,  M  is  relying  on  his  ability  to  understand  and  think  logically 
about  the  data  at  hand. 

The  task  of  remember  refers  to  M  attempting  to  "remember  a  representation  that  he  had 
already  looked  at",  "remember  a  model  from  which  to  match  or  reason",  or  "recall  a  rule  that 
matches  some  aspect  of  the  current  condition".  In  reference  to  the  general  cognitive  process 
model  this  denotes  the  process  of  retrieving  from  memory  a  relevant  representation.  Whether 
this  remembered  representation  is  constructed  via  a  concept  formation  process  or  retrieved 
directly  by  a  problem  solving  process  is  probably  dependent  on  the  situation. 

The  task  of  locate  refers  to  specifying  and  reaching  the  location  of  some  particular  element 
of  the  device.  In  Figure  2  this  refers  to  the  action  resulting  at  G.  During  this  study,  it  was 
usually  referred  to  by  M  expressing  a  simple  action  in  response  to  any  diagnosis  made  that 
required  a  test.  For  example,  upon  deciding  that  the  first  test  in  a  problem  is  to  check  the  voltage 
at  pin  D  of  connector  PI  1 1 ,  M  would  report  needing  to  "locate  or  find  Pill". 

The  tasks  of  repair  and  test  referred  to  the  physical  activities  associated  with  each.  They 
also  reflect  actions  resulting  at  G.  Examples  of  these  are  "test  Pill  for  power",  "test  a  limit 
switch  to  see  if  it  opens  and  closes",  and  "replace  a  bad  wire  in  PI  10-E". 

Closer  examination  of  the  protocols  also  shows  patterns  to  the  execution  of  these  tasks  by  M 
that  correspond  to  the  distinction  among  rule-based  and  model-based  problem  solving. 

Rule-based  processing  was  characterized  by  a  cycle  of  identify-match-locate-test  followed  by 
a  repair  or  replace  after  the  final  test.  An  example  from  the  protocols  in  problem  2  follows: 

IDENTIFY  problem  condition  -  blades  will  not  position,  no  blade  spread  light 

LOOK  in  manual  for  problem  condition 


MATCH  problem  condition  to  rule  -  find  match  at  trouble  #1 
LOCATE  element  named  in  consequent  -  blade  fold  circuit  breaker  CB80 
TEST  element  -  circuit  breaker  CB80 


(cycle  through  until  desired  test  results  obtained) 

REPAIR/REPLACE  (diagnosis  incomplete  on  this  particular  problem) 

As  shown,  the  occurrence  of  the  pattern  of  responses  that  characterizes  rule-based  processing  is 
intuitively  plausible.  Identify  produces  a  representation  that  will  be  compared  directly  to  the 
representations  of  rules  by  match.  A  similarity  judgment  might  determine  that  a  transformation 
is  needed  before  the  match  can  be  made  but  the  general  cycle  holds  up  well. 

Model-based  processing  was  characterized  by  a  cycle  of  identify -remember -match-reason- 
locate-test  followed  by  repair  or  replace  after  the  final  test.  An  example,  from  the  protocols  in 
problem  1  follows: 

IDENTIFY  problem  condition  -  blades  fold,  blades  folded  light  does  not  go  on 
REMEMBER  training  model 
MATCH  condition  to  training  model 
REMEMBER  .... 

LOCATE  slip  rings 


REASON  about  model  -  "Between  slip  ring  6  and  slip  ring  2  there  is  a  switch  for ...." 


TEST  element  -  PI  1 1-D 

(cycle  through  until  desired  test  results  obtained) 

REPAIR/REPLACE  wire 

This  pattern  of  responses  supports  the  notion  that  model-based  processing  occurs.  Identify 
produces  a  representation  that  is  compatible  with  some  aspect  of  a  functional  model  in  the  blade 
fold  system.  This  allows  M  to  deductively  reason  about  causal  relationships  concerning  those 
elements  identified  in  the  problem  condition.  As  shown,  such  reasoning  allows  M  to  reach  a 
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repair  action  in  fewer  "cognitive  steps"  than  the  rule-based  processing  allows.  This  is  desirable, 
however,  model-based  processing  also  provides  more  room  for  error.  ^ 

Information  &  Concepts 

Data.  The  data  are  the  particular  diagrams  and  pictures  M  examined,  the  reasons  he 
examined  them,  and  the  actions  he  was  carrying  out  when  examining  them. 

Findings.  As  described  earlier,  we  are  starting  with  an  abstraction  by  aggregation 
framework  for  representing  the  maintenance  object  domain.  The  problem  solving  protocols  with 
M  supported  the  use  of  that  framework  for  representation.  M  showed  a  decided  tendency  to 
move  explicitly  among  well-defined  levels  of  the  aggregation  dimension  when  closing  in  on  a 
possible  cause  of  a  problem.  He  also  used  aggregation  to  find  out  where  objects  were  located, 
what  they  were  connected  to,  and  what  they  were  parents  or  children  of.  These  relationships 
provide  an  initial  set  of  relationships  for  the  aggregation  dimension  of  the  domain  representation. 

In  both  rule-based  and  model-based  processing,  M  tended  to  start  with  a  system  level 
description  or  diagram  and  move  successively  toward  a  component  or  element  level  description 
or  diagram.  For  example,  during  rule-based  processing  on  one  problem,  M  selected  the  decision 
tree,  then  the  relays,  then  the  general  system  diagram,  and  then  a  unit  diagram  that  is  a  subset  of 
the  general  system  diagram. 

These  findings  about  the  aggregation  dimension  lend  themselves  to  the  development  of  a 
device  representation  based  on  aggregation.  Such  a  representation  will  include  objects  described 
at  a  system  level  (e.g.,  accessory-drive-system,  automatic-blade-fold- system,  auxiliary-flotation- 
system,  etc.).  Within  each  system  there  are  numerous  units  which  are  clearly  defined  (e.g.  for 
the  automatic-blade-fold-system  we  can  define  blades,  rotary-wing-head,  blade-positioning- 
system,  etc.).  Within  each  unit  there  are  numerous  components  that  are  clearly  defined  (e.g.,  for 
the  blade-positioning-system  we  can  define  blade-positioner-hydraulic-motor,  linear-actuator, 
pressure-switch-P237,  etc.).  Finally,  for  each  component  we  can  define  numerous  elements  such 
as  wire-x,  pin-y,  other-feature-z,  etc.  For  current  purposes,  the  entities  in  this  representation  can 
have  any  combination  of  the  following  relationships  with  each  other:  is-element-of,  is- 
connected-to,  is -located- at. 

3  The  w»y  to  reduce  error  and  steps  is  to  increase  skill-based  processing.  Although  we  will  not  deal  with  this  issue  here,  it  is  important 

to  note  that  the  transformation  from  model-based  and  rule -based  processing  to  skill-based  processing  is  a  transformation  of  elements 
and  relationships  into  features  that  are  directly  perceived  and  responded  to.  We  might  expect  that  the  transition  to  skill-baseD  will 
actually  be  accomplished  by  the  generation  of  rules  from  the  model  which  are  internalized  by  M  and  not  available  for  '  direct 
inspection". 
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The  actions  of  M  on  the  abstraction  dimension  are  not  as  clear  but  still  lend  support  to  the 
use  of  levels  of  abstraction  as  a  representational  form  for  the  maintainer.  Rasmussen  has  defined 
levels  of  abstraction  according  to  purpose,  function,  and  form.  Applying  his  definitions  to 
display  development  yields  diagrams  or  displays  with  specific  characteristics  at  each  level.  At 
the  purpose  level  of  abstraction,  displays  maintain  relationships  depicting  the  purpose  of 
components  within  a  larger  system,  ie.,  a  theory  of  operations  for  a  device.  At  the  function  level 
of  abstraction,  displays  maintain  relations  depicting  input-output  relationships  of  components 
with  respect  to  each  other,  ie.,  flow  diagrams  and  schematic  diagrams.  At  the  form  level  of 
abstraction,  displays  maintain  physical  relations  among  different  components  with  respect  to 
each  other,  ie.,  the  physical  look  or  location  of  a  component. 

During  the  problems  that  M  worked,  he  generally  started  at  the  level  of  form  to  identify  the 
location  of  the  element  of  interest.  However,  he  only  used  diagrams  at  a  functional  level  when 
he  was  doing  model-based  processing.  And,  he  typically  only  used  one  diagram  per  problem. 
This  could  be  an  artifact  of  the  simplicity  of  the  problems  that  M  worked.  If  problems  are 
developed  that  have  interacting  functional  systems,  then  the  maintainer  may  be  forced  to  explore 
more  system  or  device  functions  or  even  purposes  (reflected  in  theory  of  operations)  and, 
consequently,  use  multiple  levels  of  abstraction  and/or  aggregation.. 

When  M  did  use  external  representations  that  varied  on  the  abstraction  dimension,  he  seemed 
primarily  interested  in  determining  some  aspect  of  a  functional  relationship.  For  example,  he 
looked  for  what  source  provides  power  in  a  circuit  or  what  provides  a  signal  to  a  location.  These 
relationships  will  provide  an  initial  set  of  relationships  for  the  abstraction  dimension  of  the 
domain  representation.  In  particular,  we  can  represent  is-element-of,  gives -power -to,  gets- 
power-from,  gives-signal-to ,  and  gets-signal-from. 

Developing  a  representation  along  the  abstraction  dimension  requires  defining  entities  at  the 
level  of  form,  function,  and  purpose.  The  representation  described  above  is  the  representation  at 
the  level  of  form.  Now,  at  the  level  of  function,  we  can  define  entities  under  the  automatic- 
blade-fold-system  such  as  accessory-drive-control-unit,  blade-fold-circuit-schematic,  etc.  These 
functional  units  are  decomposable  through  component  levels  to  an  element  level  that  should 
define  elements  the  same  as  elements  at  the  form  level  of  representation. 

DISCUSSION 

This  study  produced  three  significant  findings  that  bear  on  the  original  hypothesis  and  the 
plan  for  Study  Two.  One  was  the  initial  cognitive  framework  derived  from  the  maintenance 
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problem  solving  protocols.  This  framework  provides  a  beginning  characterization  of  the 
cognitive  activities  required  for  maintenance  problem  solving.  With  regard  to  the  discussion  of 
the  general  cognitive  process  in  the  introduction,  it  is  particularly  interesting  to  note  the 
importance  of  several  of  these  tasks.  For  example,  several  tasks  such  as  identify,  match,  and 
reason  are  likely  to  require  a  mapping  between  some  external  representation  of  an  aspect  of  the 
problem  and  some  internal  representation  of  a  rule  or  model  or  state  knowledge  necessary  for 
problem  solving. 

Another  finding  was  the  value  of  the  aggregation  by  abstraction  space  in  describing  the 
external  representations  used  by  this  maintainer.  This  space  not  only  captured  the  information 
he  used  but  provided  differential  characterizations  according  to  particular  tasks  this  maintainer 
was  carrying  out.  For  example,  M  tended  to  use  diagrams  at  the  form  level  of  abstraction  and 
vary  level  of  aggregation  as  a  means  for  locating  components  and  systems.  It  appears  that  these 
representations  match  his  natural  representations  of  the  system  surface  features  and  physical 
relationships  very  well.  In  Study  two  we  exploited  these  tendencies  to  see  if  they  held  up  in  a 
larger  sample  of  maintained. 

In  another  example,  M  tended  to  use  diagrams  at  the  function  level  of  abstraction  and  the 
sub-system  and  circuit  levels  of  aggregation  to  determine  some  aspect  of  a  causal  relationship. 
These  were  primarily  carried  out  when  M  was  engaged  in  model-based  problem  solving.  It 
appears  that  these  levels  of  representation  are  particularly  well  matched  to  this  participant’s 
model  representations  of  the  system.  This  is  not  a  completely  surprising  result  since  the  training 
that  he  underwent  included  models  of  physical  function.  Not  all  maintained  get  this  training  — 
hence,  we  were  interested  to  see  how  well  this  finding  held  up  in  Study  Two. 

A  third  finding  was  that  model-based  problem  solving  tends  to  contain  internal  cycles  of 
rule-based  processing.  The  existence  of  these  cycles,  their  relation  to  display  use,  and  their 
relation  to  the  rest  of  the  problem  solving  process  is  difficult  to  uncover.  It  was  necessary  in 
Study  Two  to  look  for  evidence  of  this  to  further  illuminate  its  relation  to  information  use  in 
maintenance  problem  solving. 

STUDY  TWO 

OBJECTIVES 

The  primary  objective  of  this  study  was  to  develop  and  test  initial  principles  for  displays 
based  on  information  needed,  concepts  formed,  and  cognitive  processes  employed  during 
maintenance  problem  solving.  Our  general  principle  (and  experimental  hypothesis)  is  that 


13 


displays  developed  to  cover  the  abstraction  by  aggregation  information  space  will  cover  the 
range  of  possibilities  of  external  representations  needed  for  problem  solving.  Consequently,  the 
maintenance  problem  solver  should  be  able  to  find  displays  that  closely  matches  his  or  her  own 
problem  representation  and  therefore  solve  problems  more  efficiently.  The  actions  taken  are 
driven  by  the  concept  formation  process  trying  to  map  the  external  representation  to  the 
representation  needed  for  problem  solving.  The  displays  needed  are  those  examined  in  an  effort 
to  generate  the  mapping.  The  measure  of  success  for  this  test  is  the  extent  to  which  the 
maintainer  using  principled  displays  demonstrates  a  savings  in  actions  and/or  displays  needed  to 
solve  a  problem.  This  savings  would  be  over  the  actions  and/or  displays  needed  to  solve  the 
problem  using  standard  maintenance  procedures  and  diagrams  which  have  only  a  limited 
organizational  structure.4 

Another  objective  of  this  study  is  to  further  refine  the  cognitive  framework  for  maintenance 
problem  solving  begun  in  Study  One.  In  the  earlier  study  we  outlined  a  set  of  tasks  that 
maintainers  typically  engage  in  during  problem  solving.  We  expected  to  refine  this 
characterization  based  on  the  problem  solving  activities  of  this  study's  participants.  In  Study 
One  we  also  described  the  way  in  which  different  tasks  seemed  to  relate  to  information  at 
different  points  in  the  aggregation  by  abstraction  space.  Further,  we  described  the  role  of 
different  kinds  of  information  for  different  problem  solving  strategies.  In  this  study  we  wanted 
to  be  able  to  specify  whether  the  particular  relations  among  representations  of  information  (in  the 
aggregation  by  abstraction  space)  and  cognitive  activities  and  strategies  hold  up  across  several 
maintainers  and  multiple  problems.  With  regard  to  the  strategies  employed,  we  wanted  to 
further  clarify  the  mix  of  rule-based  processing  that  appears  during  predominantly  model-based 
problem  solving. 

Our  test  of  the  general  principle  and  further  refinements  to  the  cognitive  framework  were 
designed  to  provide  results  that  would  either  cause  us  to  reconsider  the  current  view  or,  yield 
insight  into  more  detailed  views  of  the  relationships  among  cognitive  activities  and  concepts 
represented  in  the  abstraction  by  aggregation  space.  If  the  general  principle  was  supported,  we 
further  planned  to  identify  such  relationships  in  this  study. 


4  It  should  be  noted  here  that  success  for  the  purpose  of  providing  small  displays  only  requires  that  maintainer  problem  solving 

performance  using  principled  materials  be  at  least  as  good  as  that  of  the  problem  solving  performance  using  standard  practices  and 


METHOD 


The  general  method  was  a  think-aloud  problem  solving  session  in  which  each  participant  was 
given  three  SH-3  bladefold  maintenance  problems  to  solve.  In  support  of  this  problem  solving 
each  participant  was  given  either  standard  materials  or  the  principled  materials  we  had 
developed.  They  were  asked  to  think  through  the  problem  aloud.  The  protocols  were  analyzed 
for  actions  taken,  display  materials  used,  strategies  employed,  and  the  relationships  among 
cognitive  activities  and  information  used. 

Participants 

The  participants  were  six  SH-3  helicopter  maintenance  technicians  in  the  U.S.  Navy.  Three 
of  these  maintained  were  relatively  inexperienced.  This  is  best  illustrated  by  the  fact  that  they 
worked  six  of  nine  problems  with  a  rule-based  problem  solving  process  (3  participants,  2  of  3 
problems  each).  The  other  three  maintained  were  relatively  experienced.  This  is  best  illustrated 
by  the  fact  that  they  worked  all  problems  with  a  model-based  problem  solving  process. 

Materials 

Pre-test  materials  consisted  of  a  verbally  presented  five  minute  explanation  and  collection  of 
demographic  information.  This  explanation  outlined  the  purpose  of  the  study  and  the  procedure 
participants  were  to  follow.  Participants  were  told  they  could  find  out  details  about  the  study 
after  completing  the  problem  solving  sessions.  The  demographic  information  was  used  to  define 
the  levels  of  experience  of  each  maintainer  in  the  study. 

There  were  several  items  in  the  problem  solving  materials.  Those  items  relevant  to  the  study 
tasks  were  the  three  SH-3  helicopter  bladefold  problems  and  the  graphical  materials  supporting 
each  problem  (standard  materials  for  one  problem  and  principled  materials  for  two  problems). 
Those  items  relevant  to  the  experimenter's  tasks  were  transcription  sheets  for  recording  actions 
taken  and  materials  examined,  a  tape  recorder  for  recording  the  problem  solving  sessions,  and 
evaluation  forms  for  the  participants  to  evaluate  the  principled  materials  they  had  used  for 
problem  solving. 

Graphical  materials  based  on  aggregation  by  abstraction  principles  were  developed  for  all 
three  of  the  problems.  The  three  levels  of  aggregation  employed  were  system,  sub-system,  and 
circuit.  In  the  SH-3  bladefold  domain,  examples  of  systems  are  BLADES  FOLD,  BLADES 
SPREAD,  and  BLADES  POSITION.  Sub-systems  of  the  BLADES  FOLD  are  BLADE- 


SEQUENCE,  BLADE-LOCK,  and  BLADE-FOLD.  A  circuit  of  the  BLADE-FOLD  sub-system 
is  the  BLADES-FOLDED  CIRCUIT.  Examples  of  each  are  shown  in  Figure  3. 


Figure  3.  Example  of  levels  of  aggregation  goes  about  here 


The  three  levels  of  abstraction  employed  were  form,  function,  and  purpose  as  defined  by 
Rasmussen  (1986).  For  these  materials,  graphics  at  the  form  level  of  abstraction  maintained  all 
the  physical  relationships  among  the  entities.  This  made  it  possible  to  see  what  the  components 
looked  like  and  where  the  components  were  located  in  relation  to  other  components  in  the  same 
system  or  circuit  at  the  same  level  of  aggregation.  Graphics  at  the  function  level  of  abstraction 
maintained  the  relationships  describing  the  physical  functions  components  carried  out  with 
respect  to  one  another.  This  made  it  possible  to  determine  how  systems  worked  and  how  they 
affected  one  another.  Graphics  at  the  purpose  level  of  abstraction  maintained  the  general 
operational  relationships  that  components  had  with  one  another.  This  made  it  possible  to 
determine  what  systems  affected  one  another  and  what  purposes  they  accomplished  in  so  doing. 
Examples  of  each  are  shown  in  Figure  4. 

Figure  4.  Example  of  levels  of  abstraction  goes  about  here 


Procedure 

We  developed  three  problems  for  the  maintainers  to  solve.  Each  problem  was  developed  in 
cooperation  with  a  subject  matter  expert  from  the  Navy.  The  expert  concluded  that  the  problems 
were  all  reasonable,  about  the  same  difficulty,  and  would  require  close  to  the  same  skills  and 
knowledge  to  solve.  We  created  supporting  display  materials  relevant  to  each  problem  at  each  of 
three  levels  of  aggregation  and  abstraction.  This  produced  a  minimum  of  nine  graphical  displays 
per  problem. 

We  presented  one  problem  to  one  maintainer  at  a  time.  Each  participant  was  asked  to  think 
aloud  as  he  worked  through  the  problem  to  solution.  Participants  were  told  they  could  work  the 
problem  using  whatever  strategies  and  procedures  they  normally  would,  with  one  constraint  —  on 
two  of  the  problems  they  could  only  use  the  display  materials  we  provided.  On  the  third 
problem,  standard  maintenance  material  was  available. 

All  participants  received  the  three  problems  in  the  same  order.  This  was  required  for  two 
reasons.  One  was  to  make  sure  that  we  could  directly  compare  different  participants  on  the  same 
problem  for  similarities  and  differences  in  actions  and  strategies.  The  other  was  to  make  sure 


these  maintainers  worked  the  standard  materials  problem  first  so  that  problem  solving  process 
would  not  be  affected  by  having  had  exposure  to  the  principled  materials.  Consequently, 
participants  worked  the  first  problem  using  standard  materials. 

After  completing  the  first  problem  using  standard  materials,  each  participant  was  taken 
through  the  problem  again  by  the  experimenter  (E).  During  this  pass,  E  used  principled  display 
materials  to  show  the  maintainer  how  such  displays  might  have  been  used  to  work  the  problem. 
This  was  done  to  familiarize  the  participant  with  the  display  concepts  they  were  going  to  see  in 
the  subsequent  problems. 

During  the  second  problem,  principled  display  material  was  available  and  the  participant  was 
required  to  work  the  problem  by  starting  with  the  material  at  the  function  level  of  abstraction. 
During  the  third  problem,  principled  display  material  was  available  and  the  participant  was 
required  to  work  the  problem  by  starting  with  the  material  at  the  purpose  level  of  abstraction.  As 
it  turned  out,  participants  primarily  moved  between  form  and  function  levels  of  abstraction. 

They  generally  found  the  purpose  level  of  abstraction  to  be  good  for  supporting  their  views  but 
not  for  developing  hypotheses  or  solutions. 

After  working  the  problems  to  solution,  each  participant  was  asked  to  make  judgments  about 
the  display  materials  they  had  used.  These  judgments  were  made  in  response  to  a  series  of 
questions  the  experimenter  asked.  The  questions  were  derived  from  Smillie,  Nugent,  Sander,  & 
Johnson  (1988). 

RESULTS 

The  results  of  this  study  are  presented  in  three  components.  The  first  considers  the 
maintenance  technicians'  protocols  to  further  refine  the  cognitive  framework  and  the  information 
space  used  in  concept  formation  by  these  maintainers.  During  this  protocol  analysis,  the  actions 
of  the  protocols  were  coded  to  be  used  for  analysis  in  the  second  component.  The  second 
component  of  the  results  takes  the  data  extracted  from  the  protocols  and  evaluates  the  displays 
used  during  the  maintenance  problem  solving  sessions.  The  third  component  of  the  results 
explores  the  relationships  among  cognitive  actions  for  concept  formation  and  problem  solving. 

Cognitive  Framework 

Data.  The  data  used  to  further  develop  the  cognitive  framework  were  the  statements  each 
maintainer  made  during  the  problem  solving  sessions  and  the  diagrams  they  examined  at  each 


step  of  problem  solving.  Each  statement  was  coded  to  reflect  a  cognitive  action  (or  more  than 
one  action)  in  which  the  maintainer  was  engaged.  The  coding  and  the  diagrams  were  used  to 
reconstruct  the  problem  solving  process  that  each  maintainer  went  through  for  each  problem 
worked. 

Findings.  Initial  analyses  of  the  Study  Two  protocols  provided  reorganization  of  the  tasks 
defined  in  Study  One.  We  identified  three  general  cognitive  processes  of  the  maintainers.  In 
addition  we  identified  five  specific  cognitive  actions  that  maintainers  carry  out  in  support  of 
those  processes.  These  processes  and  actions  are  used  to  code  the  final  versions  of  the 
participant  protocols  for  the  data  analysis  in  the  next  section.  The  processes  are: 

1 .  locate 

2.  reason 

3.  test 

The  actions  are: 

1.  look 

2.  identify 

3.  match 

4.  recall 

5.  goto 

6.  apply 

There  were  other  possible  processes  and  actions  for  which  we  have  evidence.  However,  they  did 
not  appear  often  enough  to  be  distinguished  from  one  of  the  above.  Examples  of  some  of  the 
possible  alternatives  are  trace  and  repair.  The  importance  of  these  can  only  be  determined  with 
future  studies. 

Definitions  of  most  of  these  were  provided  in  the  results  from  Study  One.  The  following 
paragraphs  will  define  those  processes  and  actions  which  are  changed  or  were  not  previously 
defined. 

Goto  refers  to  the  physical  actions  required  to  actually  change  locations.  In  reference  to  the 
general  cognitive  process  model  of  Figure  2,  this  reflects  an  action  resulting  at  G.  This  can 
occur  in  the  sense  of  the  maintainer  moving  his  or  her  body  from  one  room  to  another,  or,  it  can 
occur  in  the  sense  of  the  maintainer  moving  his  or  her  hand  from  one  page  of  a  manual  to 
another. 

Apply  refers  to  carrying  out  the  actions  embodied  by  the  use  of  some  rule,  model,  or  test 
procedure.  In  reference  to  the  general  cognitive  process  model  of  Figure  2,  this  also  reflects  an 


action  resulting  at  E.  In  these  protocols,  apply  most  often  referred  to  the  actions  of  carrying  out 
a  test  on  particular  components  of  the  system. 

Locate  has  the  same  definition  provided  in  Study  One.  It  has  now  taken  on  the  added 
significance  of  being  considered  a  higher  order  process  which  is  carried  out  by  a  set  of  actions. 
This  is  also  true  of  reason  and  test  described  in  the  following  paragraphs.  As  higher  order 
processes,  they  generate  cycles  of  action  patterns  that  would  be  represented  in  the  general 
cognitive  process  model  of  Figure  2  as  multiple  stages  with  feedback  loops.  For  example,  locate 
usually  consisted  of  a  look-identify  cycle  followed  by  a  goto  (the  stop  rule  on  the  goto  would 
typically  consist  of  an  identify-match  cycle).  The  look  results  in  visual  understanding  which 
provides  input  for  the  identify  which  results  in  concept  formation.  This,  in  turn,  provides 
feedback  to  visual  understanding  for  the  next  look. 

Reason  has  expanded  from  its  definition  in  Study  One.  It  can  refer  to  a  variety  of  cognitive 
action  patterns  which  might  characterize  a  deduce-and-conclude  strategy,  a  hypothesize-and-test 
strategy,  a  match-rule-and-fire  strategy,  an  apply-model-and-test  strategy,  or  some  other  strategy. 
In  this  study,  the  most  common  patterns  were  a  procedural  rule-firing  strategy  and  an  apply- 
model-and-generate-hypothesis  strategy.  As  such  it  usually  consisted  of  either  a  match-apply- 
recall  cycle  in  the  case  of  model  use  or  a  match-identify  cycle  in  the  case  of  rule  use. 

Test  typically  refers  to  the  process  by  which  the  problem  solver  might  apply  a  test,  identify  a 
result,  match  the  result  to  a  test  condition,  and  identify  the  next  step.  This  can  vary  depending  on 
the  particular  strategies  the  problem  solver  brings  to  the  setting.  Our  study  does  not  show  any 
vastly  different  variations  on  this  cycle. 

Examination  of  the  coded  protocols  show  support  for  the  finding  from  Study  One  that 
problem  solvers  choose  functional  level  diagrams  when  doing  model-based  reasoning.  This  was 
true  even  when  maintainers  were  given  purpose  level  diagrams  initially  and  asked  to  try  to  solve 
the  problem. 

Questioning  of  these  maintainers  showed  that  they  felt  the  purpose  level  diagrams  could  be 
used  to  support  their  views  of  how  the  system  worked.  Further,  the  diagrams  could  be  used  for 
teaching  people  how  the  system  worked.  However,  they  could  not  be  used  to  generate 
hypotheses  about  particular  physical  states  of  the  system.  Given  this,  we  expected  maintainers  to 
use  the  purpose  level  diagrams  to  generate  a  hypothesis  about  relationships  among  components 
before  going  to  function  level  diagrams  to  complete  the  problem  solution. 


Only  two  participants  formulated  hypotheses  based  on  the  purpose  level  diagrams.  Two 
others  discounted  them  altogether  while  the  final  two  chose  not  to  use  them.  It  may  turn  out  that 
purpose  level  diagrams  are  only  useful  for  a  general  process  such  as  understand.  However,  we 
saw  no  evidence  among  these  maintained  of  the  need  for  such  a  process  during  problem  solving. 

The  protocols  also  support  the  finding  from  Study  One  that  problem  solvers  use  form  level 
diagrams  along  all  levels  of  aggregation  when  locate- ing  some  system,  circuit,  or  component. 
This  was  true  of  every  maintainer.  Upon  questioning  after  the  session,  each  person  mentioned  it 
was  much  easier  to  find  locations  of  and  relations  between  systems,  sub-system,  circuits  and 
components  using  the  levels  of  aggregation  as  an  aid.  They  all  said  that  it  would  help  their  work 
tremendously  to  have  available  materials  arranged  by  level  of  aggregation  alone. 

Display  Principles  Evaluation 

Data.  The  data  were  extracted  from  the  protocol  analyses.  In  particular  we  intended  to 
examine  the  numbers  of  actions  needed  to  solve  a  problem,  the  numbers  of  looks  at  displays 
needed  to  solve  a  problem,  the  numbers  of  different  displays  needed  to  solve  a  problem,  and  the 
numbers  of  errors  committed  during  problem  solving.  It  turned  out  that  very  few  errors  were 
committed  by  these  maintainers.  Consequently,  errors  are  not  considered  here. 

The  data  used  are  briefly  defined  in  Table  2  and  described  below: 

Table  2.  Data  definitions  table  goes  about  here. 


Continued  development  of  the  cognitive  framework  yielded  two  levels  at  which  cognitive 
activities  can  be  examined.  One  level  is  that  of  cognitive  processes.  These  refer  to  general 
cognitive  processes  such  as  locate,  reason,  and  test.  So,  one  set  of  data  that  we  examined 
consisted  of  the  numbers  (frequency  counts)  of  general  cognitive  processes  each  maintainer 
employed  to  solve  a  maintenance  problem. 

The  other  level  at  which  cognitive  activities  can  be  examined  is  that  of  cognitive  actions 
which  make  up  processes.  The  general  cognitive  processes  are  made  up  of  specific  cognitive 
actions  such  as  match,  identify,  and  look.  As  discussed  earlier,  there  is  not  a  fixed  number  of 
actions  per  process;  instead  there  may  be  different  actions  at  different  times  or  there  may  be 
recursive  actions.  Consequently,  the  measures  of  process  and  actions  indicate  different  levels  of 


effort  involved  in  the  problem  solving  process.5  The  second  set  of  data  we  examined  consisted 
of  the  numbers  (frequency  counts)  of  the  specific  cognitive  actions  each  maintainer  employed  to 
solve  a  maintenance  problem.  The  third  set  of  data  we  examined  were  the  numbers  and  kinds  of 
displays  examined  by  the  maintainers  during  their  problem  solving  sessions. 

The  following  analyses  compare  the  numbers  of  cognitive  processes,  cognitive  actions,  and 
displays  examined  to  solve  maintenance  problems.  This  presented  a  problem  since  we  wanted  to 
compare  results  from  one  problem  to  another.  Each  problem  had  a  different  number  of 
processes,  actions,  and  displays  that  were  needed  according  to  standard  procedures  to  obtain  a 
solution.  To  account  for  this  difference  we  transformed  each  raw  frequency  into  a  proportion  of 
the  baseline  actions  that  were  needed  to  solve  that  particular  problem  according  to  standard 
procedures  in  the  troubleshooting  manual. 

To  create  baselines  for  each  problem,  we  worked  each  problem  according  to  the 
troubleshooting  chart  in  the  SH-3  maintenance  manual.  We  transcribed  every  action  and  display 
that  was  required  to  solve  the  problem  according  to  the  manual.  We  then  divided  each  raw 
frequency  score  by  the  baseline  to  produce  a  proportion  of  baseline  score.  For  example, 
Participant  1  employed  13  cognitive  processes  to  solve  the  first  problem  using  standard 
materials.  The  baseline  number  of  cognitive  processes  needed  for  the  first  problem  was  16.  This 
yielded  a  proportion  of  .81.  Participant  1  was  able  to  solve  the  first  maintenance  problem  using 
the  experimental  displays  by  resorting  to  only  81%  of  the  number  of  cognitive  processes  they 
would  have  used  with  standard  procedures. 

Findings.  These  findings  assess  the  value  of  the  principled  displays,  in  general,  as  support 
for  the  maintenance  problem  solving  process.  It  is  important  to  remember  that  we  only 
examined  data  from  six  maintainers.  Consequently,  the  results  from  the  following  ANOVAS  are 
based  on  a  small  N. 

There  were  four  analyses  conducted  to  evaluate  how  well  principled  displays  served  to 
support  the  problem  solving  process  of  these  maintainers.  Each  was  an  ANOVA  comparing  the 
mean  proportion  of  baseline  scores  for  principled  material  presentation  to  the  mean  proportion  of 
baseline  scores  for  standard  materials  presentations.  This  is  the  MATERIALS  variable.  It  has 
three  levels  -  1)  standard  materials;  2)  principled  materials,  function  level  seen  first;  3) 
principled  materials,  purpose  level  seen  first.  The  other  grouping  variable  is  EXPERIENCE  -  1) 
experienced;  2)  not  experienced.  The  mean  scores  by  variables  for  cognitive  processes, 
cognitive  actions,  and  displays  examined  are  listed  in  Table  3. 


5  This  m*y  have  implications  for  work  such  as  the  cognitive  toad  work  of  Swcller  (1988). 


Table  3.  Mean  scores  goes  about  here. 


The  first  analysis  examined  the  value  of  these  displays  for  decreasing  the  frequency  with 
which  general  cognitive  processes  were  employed  in  problem  solving.  We  analyzed  the  scores 
according  to  experience  level  and  principled  material  presentation  (EXPERIENCE  x 
MATERIAL;  2x3  ANOVA).  This  analysis  produced  significant  main  effects  but  no  interaction 
effects.  The  mean  proportion  of  baseline  scores  for  general  cognitive  processes  used  when 
problem  solving  with  principled  materials  are  significantly  different  from  the  proportion  of 
baseline  scores  when  using  standard  materials  [F(2 12)  =  5.038;  p  =  .026].  Figure  5  illustrates  this 
difference  and  shows  that  the  decrease  in  frequency  of  use  of  cognitive  processes  is  greater  for 
principled  materials  than  for  standard  materials.  This  analysis  also  showed  the  mean  proportion 
of  baseline  scores  for  experienced  maintainers  (M  =  .422)  tend  to  be  better  than  the  mean 
proportion  of  baseline  scores  of  inexperienced  maintainers  [M  =  .907;  F(1  ]2)  =  15.122;  p  =  .002], 
These  maintainers,  when  using  principled  materials,  used  general  cognitive  processes  only  .40  to 
.71  as  often  as  required  by  baseline.  When  using  standard  materials,  these  maintainers  use 
general  cognitive  processes  .88  as  often  as  the  baseline  procedures  required. 

Figure  5.  Graph  comparing  the  means  of  materials  by  experience  level  for 
cognitive  processes  goes  about  here. 


The  second  analysis  examined  the  value  of  these  displays  for  saving  on  the  number  of 
specific  cognitive  actions  employed  in  problem  solving.  We  analyzed  the  scores  according  to 
experience  level  and  principled  material  presentation  (EXPERIENCE  x  MATERIAL;  2x3 
ANOVA).  This  analysis  produced  a  significant  main  effect  for  materials  only.  For  specific 
cognitive  actions,  the  mean  score  when  using  principled  display  materials  is  significantly 
different  across  types  of  material  used  [F(212)  =  13.623;  p  =  .004],  Figure  6  illustrates  this 
difference  and  shows  that  the  savings  over  baseline  are  greater  for  principled  materials  than  they 
are  for  standard  materials.  These  maintainers,  when  using  principled  materials,  required  only  .42 
to  .76  of  the  cognitive  actions  required  in  the  baseline.  When  using  standard  materials,  these 
maintainers  used  1.40  times  as  many  cognitive  actions  as  the  baseline. 

Figure  6.  Graph  comparing  the  means  of  materials  by  experience  level  for 
cognitive  actions  goes  about  here. 


The  third  analysis  examines  the  value  of  these  displays  for  saving  on  the  number  of  displays 
examined  during  maintenance  problem  solving.  We  analyzed  the  mean  proportion  of  baseline 
scores  according  to  experience  level  and  principled  material  presentation  (EXPERIENCE  x 


MATERIAL;  2x3  ANOVA).  This  analysis  produced  a  significant  main  effect  for  material 
presentation  and  an  interaction  effect  for  experience  x  material.  The  mean  proportion  of  baseline 
scores  for  displays  examined  are  significantly  different  across  types  of  material  used  [F(2.i2)  = 
18.491;  p  <  .001].  Figure  7  illustrates  this  difference  and  shows  that  the  savings  over  baseline 
are  greater  when  examining  principled  display  materials  than  they  are  when  examining  standard 
display  materials.  These  maintainers,  when  using  principled  materials,  required  .25  to  .45  as 
many  displays  as  the  baseline.  When  using  standard  display  materials,  they  required  1.60  times 
as  many  displays  as  the  baseline. 

Figure  7.  Graph  comparing  the  means  of  materials  by  experience  level  for 
displays  examined  goes  about  here. 


The  interaction  effect  reflects  a  performance  difference  between  experienced  and 
inexperienced  maintainers  across  material  presentation  [F(2i]2)  =  5.334;  p  =  .022].  Experienced 
maintainers  examined  more  displays  relative  to  baseline  than  inexperienced  maintainers  when 
using  standard  materials.  However,  ''  hen  using  principled  materials,  experienced  maintainers 
were  much  more  efficieiu  ...u  snowed  a  greater  improvement  over  baseline  than  inexperienced 
maintainers.  Although  this  interaction  is  interesting,  it  is  difficult  to  interpret  given  the  small  N 
for  the  study.  One  possibility  suggests  this  is  due  to  the  fact  that  experienced  people  do  not 
normally  use  standard  materials.  It  will  be  necessary  to  examine  this  in  larger  studies. 

These  three  analyses  provide  strong  support  for  the  use  of  displays  developed  according  to 
the  use  of  the  aggregation  by  abstraction  principles.  It  appears  that  materials  developed  to  match 
the  level  of  representation  at  which  maintenance  problem  solvers  are  working  enable  significant 
savings  in  the  actions  carried  out  and  the  displays  examined  during  maintenance  problem 
solving.  It  is  tempting  to  think  that  this  difference  might  be  due  to  greater  reliance  on  model- 
based  problem  solving  in  the  use  of  principled  materials.  However,  each  of  the  experienced 
maintainers  use  model-based  processing  with  the  standard  materials  presentation.  Comparison 
of  the  mean  savings  scores  on  the  standard  materials  presentation  for  experienced  vs. 
inexperienced  maintainers  from  Table  3  shows  no  difference.  This  indicates  that  model-based 
problem  solving  was  not  the  key  factor  in  producing  the  savings  in  cognitive  activities. 
Consequently,  the  displays  supported  problem  solving  across  strategies,  processes,  actions,  and 
particular  displays. 


Concept  Formation  in  Pisplav-based  Problem  Solving 


Data.  These  are  the  same  data  as  used  in  the  previous  analyses;  however,  there  is  a  different 
focus.  These  data  focus  on  the  numbers  of  different  kinds  of  actions  taken  when  using  different 
display  materials  for  solving  problems.  Consequently  the  findings  are  presented,  for  each  action, 
as  the  proportion  of  baseline  actions  for  a  particular  problem  associated  with  particular  display 
material.  The  analysis  examines  the  relationships  among  these  actions,  the  role  they  have  in 
concept  formation,  and  the  differential  effects  they  might  have  upon  maintenance  problem 
solving. 

Findings.  The  findings  illustrate  the  way  in  which  different  cognitive  actions  occur  for 
problems  in  which  different  kinds  of  displays  are  examined.  We  assume  that  any  differences 
found  reflect  differences  in  the  underlying  concept  formation  process  contributing  to 
maintenance  problem  solving.  To  establish  whether  a  difference  existed  we  conducted  a 
MATERIAL  x  ACTIONS  ANOVA  (3  x  6)  on  the  mean  proportion  of  baseline  as  described 
previously.  This  analysis  showed  significant  differences  among  ACTIONS  [F(5  i90)  =  8.210;  p  < 
.001]  for  the  number  of  actions  used  relative  to  baseline  as  well  as  among  MATERIALS  [F(290) 

=  18.174;  p  <  .001].  There  was  no  ACTIONS  x  MATERIALS  interaction.  Since  the  interaction 
was  not  significant  we  only  examined  the  inter-MATERIAL  differences  shown  in  Figure  8  for 
trends  and  insights  relative  to  concept  formation  and  maintenance  problem  solving. 

Figure  8.  Graph  comparing  the  means  of  materials  by  specific  actions  for 
displays  examined  goes  about  here. 


The  general  pattern  of  differences  in  actions  required  for  solving  these  maintenance  problems 
is  about  the  same  across  all  the  actions  for  all  display  materials  used.  Interesting  in  the  general 
pattern  is  that,  relative  to  baseline,  maintainers  engage  in  more  identify  actions  when  solving 
problems  with  standard  materials  than  they  do  when  solving  problems  with  materials  supporting 
form,  functions,  and  purpose  levels  of  abstraction  relevant  to  the  problem.  This  trend  of  more 
actions  required  relative  to  baseline  for  using  standard  materials  holds  up  across  all  actions. 
However,  the  amount  of  change  in  actions  required  from  standard  materials  to  principled 
materials  appears  to  be  different  for  identify,  match,  and  look.  For  example,  on  match  the  mean 
proportion  of  baseline  actions  goes  from  .21  to  .13  (standard  materials  to  function-first  materials) 
for  a  change  of  .8  while  the  change  is  .21  for  look.  These  differences  suggest  differences  in  the 
way  aggregation  by  abstraction  based  displays  support  concept  formation  and  problem  solving. 
The  rest  of  this  section  addresses  patterns  that  appear  relevant  to  these  differences. 


One  general  pattern  is  the  decrease  in  actions  going  from  standard  displays  to  function-first 
displays  to  purpose-first  displays  as  measured  across  all  actions.  Viewed  as  a  whole,  this  result 
provides  support  for  the  view  that  these  principled  displays  benefit  visual  understanding,  concept 
formation,  and  problem  solving  as  defined  in  Figure  2  and  discussed  in  previous  results.  This 
finding  might  be  confounded  by  a  practice  effect,  therefore  it  illustrates  several  research  issues  to 
be  clarified.  One  is  whether  the  improvement  from  function-first  materials  to  purpose-first 
materials  represents  improved  processing  support  or  improved  display  use  due  to  practice.  A 
second  question  is  whether  support  from  the  displays  is  specific  to  particular  action  or  actions. 
Third  is  whether  improvements  in  a  particular  action  propagates  improvements  elsewhere;  or, 
whether  actions  are  supported  directly  by  the  displays  without  propagating  effects  to  other 
actions. 

Another  pattern  is  the  slightly  greater  improvement  for  identify  over  all  other  actions  except 
look.  Since  identify  corresponds  to  concept  formation  of  a  relevant  internal  representation  then 
the  data  suggest  that  these  principled  displays  do  more  to  support  concept  formation  than  for  all 
other  processes  except  visual  understanding.  This  is  illustrated  in  the  differences  in  Figure  8. 

This  finding  suggests  that  with  display  materials  developed  according  to  the  aggregation  by 
abstraction  principle,  fewer  looks  are  required  to  find  useful  material  from  which  relevant 
internal  representations  can  be  formed.  And  it  appears  that  some  of  the  diagrams  may  provide 
exactly  the  representation  needed  by  maintained  for  problem  solving,  without  having  to 
transform  the  representation  through  concept  formation. 

A  final  noticeable  pattern  emerges  from  examining  improvements  in  recall ,  identify,  and 
match  as  compared  to  look.  The  improvement  in  look  is  dramatic  while  the  improvement  in  the 
other  three  is  moderate.  This  could  happen  for  two  possible  reasons.  Decreased  numbers  of  look 
might  force  the  problem  solver  to  depend  more  heavily  on  other  actions  than  they  would  have 
otherwise.  This  interpretation  means  that  look  actions  constrain  the  use  of  other  actions.  The 
other  possible  reason  is  that  increased  numbers  of  recall,  identify,  and  match  cause  there  to  be 
less  need  for  look.  This  interpretation  means  that  other  actions  enable  look.  This  distinction 
cannot  be  resolved  with  these  data  and  will  have  to  be  addressed  in  future  studies. 

DISCUSSION 

This  study  produced  three  interesting  sets  of  findings  that  bear  on  the  original  hypothesis. 

One  was  the  refinement  of  the  cognitive  framework  for  maintenance  problem  solving.  Asa 
result  of  this  study  we  revised  the  framework  to  incorporate  cognitive  processes  and  actions 
comprising  those  processes.  During  this  refinement  we  identified  a  number  of  different  cycles  of 


actions  that  carry  out  the  same  processes.  It  appears  that  these  different  cycles  might  be  related 
to  different  overarching  problem  solving  strategies  and  the  use  of  information  represented 
differently  in  the  abstraction  by  aggregation  space.  This  requires  further  analysis  and  study  that 
will  be  carried  out  at  a  later  date.  Our  framework  and  methods  provide  the  tools  for  subsequent 
data  collection  and  analyses. 

The  second  set  of  findings  demonstrated  the  value  of  using  diagrams  developed  according  to 
principles  of  aggregation  by  abstraction  for  maintenance  problem  solving.  In  every  analysis, 
either  one  or  both  sets  of  principled  materials  demonstrated  a  significant  savings  in  processes, 
actions,  and  displays  used  over  standard  materials  using  manual-dictated  procedures.  This 
means  that  principled  materials  provided  savings  for  general  cognitive  processes  carried  out, 
specific  cognitive  actions  carried  out,  and  for  display  materials  examined  during  the  course  of 
maintenance  problem  solving.  These  savings  were  even  demonstrated  by  maintainers  using 
principled  display  materials  with  manual-dictated  procedures  (i.e.,  use  the  troubleshooting 
charts). 

The  third  set  of  findings  demonstrated  the  differential  effect  of  aggregation  by  abstraction 
based  displays  on  cognitive  actions,  in  particular  concept  formation.  The  findings  indicate  the 
value  of  these  kinds  of  external  representations  for  identifying  a  relevant  internal  representation, 
matching  a  representation  to  some  element  of  the  problem  space  such  as  a  rule  or  model,  and 
remembering  a  relevant  internal  representation.  In  particular  the  findings  revealed  three 
questions  that  need  to  be  answered  in  future  work.  One  question  concerns  the  extent  to  which 
purpose-first  displays  supported  problem  solving  over  other  displays  versus  how  much  practice 
effect  there  was  for  the  use  of  principled  displays.  Another  question  is  how  much  support  the 
aggregation  by  abstraction  based  displays  give  to  a  particular  cognitive  process  such  as  concept 
formation  versus  being  a  general  support  for  all  cognitive  processes  involved  in  maintenance 
problem  solving.  The  third  question,  related  to  the  second,  is  whether  the  effect  of  these  displays 
is  to  enable  some  actions  as  a  result  of  using  other  actions;  or,  to  constrain  some  actions  as  a 
result  of  changes  in  others. 

Future  studies  and  analyses  will  explore  the  relationships  of  the  principled  displays  to  the 
particular  actions  and  particular  concept  representations  that  maintainers  employ  during  problem 
solving. 


CONCLUSIONS 


There  were  two  general  goals  to  this  work.  One  was  to  develop  principles  to  guide  the 
transformation  of  hardcopy  formats  of  qualitative  and  quantitative  information  to  electronically 
displayed  formats.  For  these  studies,  the  goal  was  scaled  down  to  the  development  of  initial 
principles  that  could  be  tested  on  a  small  scale.  The  other  goal  was  to  evaluate  these  principles 
in  a  realistic  context  with  maintainers  of  an  operational  system.  The  evaluation  was  aimed  at 
determining  whether  to  continue  developing  the  principles  based  on  our  current  view  or  whether 
to  re-evaluate  our  current  view. 

Toward  the  first  goal  we  developed  a  view  of  the  cognition  required  for  maintenance 
problem  solving.  A  central  aspect  of  this  view  is  that  the  key  cognitive  activity  for  our  purposes 
is  a  concept  formation  process  which  maps  external  representations  such  as  diagrams  to  relevant 
internal  representations  such  as  rules  or  models.  Furthermore,  we  proposed  a  structure-mapping 
mechanism  as  the  means  by  which  concept  formation  takes  place.  This  structure-mapping 
transforms  a  base  representation  to  a  target  representation  by  making  similarity  judgments  along 
levels  of  aggregation  (features)  and  abstraction  (relations  among  features).  As  a  general 
principle,  we  proposed  that  displays  to  support  maintenance  problem  solving  should  vary  along 
these  dimensions.  To  the  extent  there  is  a  variety  of  display  representations,  the  maintainer 
would  be  more  likely  to  match  the  right  level  of  representation  to  specific  processing  needs. 

The  initial  study  supported  this  view  and  provided  the  opportunity  to  develop  a  framework  of 
the  cognitive  activities  of  the  maintainer.  To  achieve  the  second  goal,  a  second  study  was 
conducted  in  which  principled  displays  were  used  by  the  maintainer  for  problem  solving 
scenarios.  The  results  for  the  use  of  principled  displays  were  compared  to  those  for  using 
standard  materials.  The  use  of  principled  displays  led  to  a  significant  savings  in  actions 
engaged,  as  coded  from  the  framework,  and  displays  examined  by  the  maintainers. 

Use  of  the  aggregation  by  abstraction  information  space  as  a  way  of  operationally  defining 
forms  of  external  representation  holds  promise  for  the  development  of  displays  supporting 
maintenance  problem  solving.  Work  in  the  next  phase  of  this  project  will  further  operationalize 
the  definitions  of  the  different  levels  of  aggregation  and  abstraction.  Based  on  this 
operationalization  and  further  understanding  of  the  specific  maintenance  domain  we  will  develop 
a  wide  range  of  displays  that  should  be  useful  to  a  broader  range  of  maintenance  problems.  This 
should  allow  us  to  explore  not  only  the  utility  of  displays  based  on  »he  concepts  but  also  the 
relation  of  the  displays  and  the  information  they  contain  to  the  conceptual  representations  of  the 
maintainers. 


This  work  showed  that  maintenance  problem  solvers  engage  in  a  clearly  definable  set  of 
processes  and  actions.  These  processes  can  be  examined  for  the  way  in  which  they  affect  or  are 
affected  by  information  in  external  displays.  So  the  next  phase  of  this  work  will  also  examine  in 
more  detail  the  relationships  of  the  aggregation  by  abstraction  space  to  cognitive  processing  in 
maintenance  problem  solving.  In  particular  we  expect  to  explore  how  the  match  and  identify 
cognitive  actions  are  carried  out  We  expect  to  see  that  they  occur  as  concept  formation  and  can 
be  modelled  by  a  structure-mapping  mechanism.  Using  this  model  should  allow  us  to  develop 
and  predict  the  effects  of  good  and  poor  external  representations. 

Plans  for  the  future  include  a  series  of  rigorous,  controlled  experiments  which  will  explore 
the  issues  discussed  above.  If  the  promise  demonstrated  by  the  aggregation  by  abstraction 
information  space  for  supporting  concept  formation  holds  up  in  more  rigorous  and  complex 
studies  then  it  should  not  be  long  before  a  solid  set  of  principles  is  available  to  the  development 
of  small  displays  for  maintenance  problem  solving. 
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TABLES 


Table  1 


Table  2 


Data  Definitions. 


Dependent  Variable 

Definition 

number  of  cognitive  processes 

sum  of  the  frequencies  of  occurrence  of  reason,  locate,  test 
within  each  problem  for  each  subject 

number  of  cognitive  actions 

sum  of  the  frequencies  of  occurrence  of  apply,  goto,  identify, 
look,  match,  recall  within  each  problem  for  each  subject 

number  of  displays  examined 

sum  of  the  frequencies  of  displays  examined  within  each 
problem  for  each  subject 

Table  3 


Mean  Proportion  of  Baseline  Scores  for  Processes,  Actions,  and  Displays  Examined. 
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Figure  Caption 


Figure  1.  The  maintenance  domain. 
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Figure  Caption 


Figure  2.  The  general  cognitive  process. 


Figure  Caption 


Figure  3.  Example  of  levels  of  aggregation. 
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Figure  Caption 


Figure  4.  Example  of  levels  of  abstraction. 


Figure  Caption 


Figure  5.  Comparing  the  mean  proportion  of  baseline  scores  for  general  cognitive  processes 
across  materials  and  experience. 
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Figure  Caption 


Figure  6.  Comparing  the  mean  proportion  of  baseline  scores  for  cognitive  actions  across 
materials  and  experience. 
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Figure  Caption 


Figure  7.  Comparing  the  mean  proportion  of  baseline  scores  for  displays  examined 
materials  and  experience. 
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Figure  Caption 


Figure  8.  Comparing  the  means  of  materials  by  specific  actions  for  displays  examined. 
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